Pandas DataFrame | shift method
Start your free 7-days trial now!
Pandas DataFrame.shift(~)
method shifts the rows or columns by the specified amount, without changing the size of the DataFrame.
Parameters
1. periods
| int
The amount to shift by. If axis=0
(the default), then positive integers mean rows are shifted downwards, while negative integers mean rows are shifted upwards. The same logic applies for when axis=1
(column shifts).
2. freq
link | string
or DataOffset
or tseries.offsets
or timedelta
| optional
If specified, then the date index will be shifted instead of rows/columns. This is only relevant if the index of the source DataFrame is a DatetimeIndex
. Check out the examples below for clarification.
3. axis
| int
or string
| optional
Whether to shift rows or columns:
Axis | Description |
---|---|
| Rows will be shifted. |
| Columns will be shifted. |
By default, axis=0
.
4. fill_value
link | object
| optional
The value used to fill the newly shifted rows/columns. The default fill_value
depends on the column's data type:
For date-related types,
fill_value=NaT
.Otherwise,
fill_value=NaN
.
Return Value
A new DataFrame
with the index, rows or columns shifted.
Examples
Consider the following DataFrame:
df = pd.DataFrame({"A":["a","b","c","d"],"B":["e","f","g","h"],"C":["i","j","k","l"]})df
A B C0 a e i1 b f j2 c g k3 d h l
Basic usage
To shift down by 2 rows:
df.shift(2)
A B C0 NaN NaN NaN1 NaN NaN NaN2 a e i3 b f j
To shift up by 2 rows:
df.shift(-2)
A B C0 c g k1 d h l2 NaN NaN NaN3 NaN NaN NaN
Notice how the index did not shift, that is, the index is still [0,1,2,3]
.
Shifting columns
By default, axis=0
, which means that shifting happens for rows. To shift columns, pass axis=1
like so:
df.shift(2, axis=1)
A B C0 NaN NaN a1 NaN NaN b2 NaN NaN c3 NaN NaN d
Again, notice how the column values have shifted, but not the column labels.
Specifying fill_value
Instead of filling with the default NaN
, we can specify our very own fill value like so:
df.shift(fill_value=0)
A B C0 0 0 01 0 0 02 a e i3 b f j
Specifying freq
The freq
parameter comes into play when the DataFrame is a time-series.
To demonstrate, consider the following time-series DataFrame:
index_date = pd.date_range("2020-12-25", periods=4, freq="D")df = pd.DataFrame({"A":["a","b","c","d"],"B":["e","f","g","h"],"C":["i","j","k","l"]}, index=index_date)df
A B C2020-12-25 a e i2020-12-26 b f j2020-12-27 c g k2020-12-28 d h l
To shift the index by 3 days:
df.shift(freq="3D")
A B C2020-12-28 a e i2020-12-29 b f j2020-12-30 c g k2020-12-31 d h l
Notice how the dates in the index have been shifted by 3 days, yet the values (e.g. a
, b
and so on) have not shifted at all.
You can also combine period
and freq
. To shift the index by 2*3=6 days
:
df.shift(2, freq="3D")
A B C2020-12-31 a e i2021-01-01 b f j2021-01-02 c g k2021-01-03 d h l
Unintuitive behaviour
Consider the following DataFrame:
index_date = pd.date_range("2020-12-29", periods=4, freq="D")df = pd.DataFrame({"A":["a","b","c","d"],"B":["e","f","g","h"],"C":["i","j","k","l"]}, index=index_date)df
A B C2020-12-29 a e i2020-12-30 b f j2020-12-31 c g k2021-01-01 d h l
Setting freq="M"
yields:
df.shift(periods=1, freq="M")
A B C2020-12-31 a e i2020-12-31 b f j2021-01-31 c g k2021-01-31 d h l
The results is slightly unintuitive since we'd expect an one-month offset to be applied to each date index.
Note the following:
each date index got shifted to the end of the month.
since we specified
freq="M"
, the day unit of the date index becomes irrelevant.the exception is when the day unit is at the end of the month (e.g.
12-31
), in which case the month will get shifted (e.g. to01-31
).