search
Search
Login
Unlock 100+ guides
menu
menu
web
search toc
close
Comments
Log in or sign up
Cancel
Post
account_circle
Profile
exit_to_app
Sign out
What does this mean?
Why is this true?
Give me some examples!
search
keyboard_voice
close
Searching Tips
Search for a recipe:
"Creating a table in MySQL"
Search for an API documentation: "@append"
Search for code: "!dataframe"
Apply a tag filter: "#python"
Useful Shortcuts
/ to open search panel
Esc to close search panel
to navigate between search results
d to clear all current filters
Enter to expand content preview
icon_star
Doc Search
icon_star
Code Search Beta
SORRY NOTHING FOUND!
mic
Start speaking...
Voice search is only supported in Safari and Chrome.
Navigate to

Pandas DataFrame | shift method

schedule Aug 12, 2023
Last updated
local_offer
PythonPandas
Tags
mode_heat
Master the mathematics behind data science with 100+ top-tier guides
Start your free 7-days trial now!

Pandas DataFrame.shift(~) method shifts the rows or columns by the specified amount, without changing the size of the DataFrame.

NOTE

Use this method in favour of tshift(~), since tshift(~) is now deprecated.

Parameters

1. periods | int

The amount to shift by. If axis=0 (the default), then positive integers mean rows are shifted downwards, while negative integers mean rows are shifted upwards. The same logic applies for when axis=1 (column shifts).

2. freqlink | string or DataOffset or tseries.offsets or timedelta | optional

If specified, then the date index will be shifted instead of rows/columns. This is only relevant if the index of the source DataFrame is a DatetimeIndex. Check out the examples below for clarification.

3. axis | int or string | optional

Whether to shift rows or columns:

Axis

Description

0 or "index"

Rows will be shifted.

1 or "columns"

Columns will be shifted.

By default, axis=0.

4. fill_valuelink | object | optional

The value used to fill the newly shifted rows/columns. The default fill_value depends on the column's data type:

  • For date-related types, fill_value=NaT.

  • Otherwise, fill_value=NaN.

Return Value

A new DataFrame with the index, rows or columns shifted.

Examples

Consider the following DataFrame:

df = pd.DataFrame({"A":["a","b","c","d"],"B":["e","f","g","h"],"C":["i","j","k","l"]})
df
A B C
0 a e i
1 b f j
2 c g k
3 d h l

Basic usage

To shift down by 2 rows:

df.shift(2)
A B C
0 NaN NaN NaN
1 NaN NaN NaN
2 a e i
3 b f j

To shift up by 2 rows:

df.shift(-2)
A B C
0 c g k
1 d h l
2 NaN NaN NaN
3 NaN NaN NaN

Notice how the index did not shift, that is, the index is still [0,1,2,3].

Shifting columns

By default, axis=0, which means that shifting happens for rows. To shift columns, pass axis=1 like so:

df.shift(2, axis=1)
A B C
0 NaN NaN a
1 NaN NaN b
2 NaN NaN c
3 NaN NaN d

Again, notice how the column values have shifted, but not the column labels.

Specifying fill_value

Instead of filling with the default NaN, we can specify our very own fill value like so:

df.shift(fill_value=0)
A B C
0 0 0 0
1 0 0 0
2 a e i
3 b f j

Specifying freq

The freq parameter comes into play when the DataFrame is a time-series.

To demonstrate, consider the following time-series DataFrame:

index_date = pd.date_range("2020-12-25", periods=4, freq="D")
df = pd.DataFrame({"A":["a","b","c","d"],"B":["e","f","g","h"],"C":["i","j","k","l"]}, index=index_date)
df
A B C
2020-12-25 a e i
2020-12-26 b f j
2020-12-27 c g k
2020-12-28 d h l

To shift the index by 3 days:

df.shift(freq="3D")
A B C
2020-12-28 a e i
2020-12-29 b f j
2020-12-30 c g k
2020-12-31 d h l

Notice how the dates in the index have been shifted by 3 days, yet the values (e.g. a, b and so on) have not shifted at all.

You can also combine period and freq. To shift the index by 2*3=6 days:

df.shift(2, freq="3D")
A B C
2020-12-31 a e i
2021-01-01 b f j
2021-01-02 c g k
2021-01-03 d h l

Unintuitive behaviour

Consider the following DataFrame:

index_date = pd.date_range("2020-12-29", periods=4, freq="D")
df = pd.DataFrame({"A":["a","b","c","d"],"B":["e","f","g","h"],"C":["i","j","k","l"]}, index=index_date)
df
A B C
2020-12-29 a e i
2020-12-30 b f j
2020-12-31 c g k
2021-01-01 d h l

Setting freq="M" yields:

df.shift(periods=1, freq="M")
A B C
2020-12-31 a e i
2020-12-31 b f j
2021-01-31 c g k
2021-01-31 d h l

The results is slightly unintuitive since we'd expect an one-month offset to be applied to each date index.

Note the following:

  • each date index got shifted to the end of the month.

  • since we specified freq="M", the day unit of the date index becomes irrelevant.

  • the exception is when the day unit is at the end of the month (e.g. 12-31), in which case the month will get shifted (e.g. to 01-31).

robocat
Published by Isshin Inada
Edited by 0 others
Did you find this page useful?
thumb_up
thumb_down
Comment
Citation
Ask a question or leave a feedback...
thumb_up
0
thumb_down
0
chat_bubble_outline
0
settings
Enjoy our search
Hit / to insta-search docs and recipes!