Describe the bug I have setup a systemd service to monitor the start of Twitter spaces, and the time interval of running the command is 20 seconds. I do have multiple services monitoring multiple accounts, so it's normal to see some 429 Too Many Requests returned from the Twitter API. However, the response content didn't seem to be valid JSON (likely HTML), and it would cause a JSONDecodeError to be raised.

To Reproduce

twspace_dl --input-cookie-file "$twitter_cookies" -suU "https://twitter.com/$username" -o '/tmp/twspace_dl-%(creator_screen_name)s' -m -v

Expected behavior It's not a big issue, but I think it would be better if this can be handled properly. Maybe retry after a timeout, or a better error message?

Output (Note: username and user ID are replaced by placeholders in the following log output)

2022-06-20 09:02:39,132 [DEBUG] Starting new HTTPS connection (1): cdn.syndication.twimg.com:443
2022-06-20 09:02:39,148 [DEBUG] https://cdn.syndication.twimg.com:443 "GET /widgets/followbutton/info.json?screen_names=<username> HTTP/1.1" 200 178
2022-06-20 09:02:39,149 [DEBUG] Starting new HTTPS connection (1): twitter.com:443
2022-06-20 09:02:39,360 [DEBUG] https://twitter.com:443 "GET /i/api/fleets/v1/avatar_content?user_ids=<user_id>&only_spaces=true HTTP/1.1" 429 0
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 525, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/twspace_dl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/twspace_dl/__main__.py", line 199, in main
    args.func(args)
  File "/usr/local/lib/python3.9/dist-packages/twspace_dl/__main__.py", line 63, in space
    twspace = Twspace.from_user_avatar(args.user_url, auth_token)
  File "/usr/local/lib/python3.9/dist-packages/twspace_dl/twspace.py", line 231, in from_user_avatar
    avatar_content = requests.get(
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0

Desktop (please complete the following information):

• OS: Linux
• Version: Ubuntu 21.10
• Installation method: pip

strftime() Format Codes didn't have %s and maybe means %f Microsecond as a decimal number, zero-padded to 6 digits. ? https://github.com/Ryu1845/twspace-dl/blob/daa6ab1d1fdf5f56f4b6d95717a5c0295d8b50cc/twspace_dl/main.py#L35 And this is the error that use %s

Traceback (most recent call last):
  File "c:\users\test01\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\test01\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\test01\AppData\Local\Programs\Python\Python39\Scripts\twspace_dl.exe\__main__.py", line 7, in <module>
  File "c:\users\test01\appdata\local\programs\python\python39\lib\site-packages\twspace_dl\__main__.py", line 199, in main
    args.func(args)
  File "c:\users\test01\appdata\local\programs\python\python39\lib\site-packages\twspace_dl\__main__.py", line 34, in space
    log_filename = datetime.datetime.now().strftime(
ValueError: Invalid format string

Btw, the log file only write the last message like

2022-03-14 01:15:13,245 [INFO] Finished downloading

or

2022-03-14 01:17:14,383 [ERROR] Can't Download. Space has ended, can't retrieve master url. You can provide it with -f URL if you have it.

but in cmd screen even have like

2022-03-14 01:34:35,215 [INFO] ./[2022-03-12]creator_screen_name-id.m3u8 written to disk
2022-03-14 01:34:35,955 [INFO] .\tmppnmz5g9v\[2022-03-12]creator_screen_name-id.m3u8 written to disk

doesn't the two lines need write in log file, too?

Describe the bug https://github.com/Ryu1845/twspace-dl/blob/1ca12aa975c40c5d9020709e22332b041b2f13ef/twspace_dl/main.py#L173-L176

To Reproduce twspace_dl -f https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/YRSsw6_P5xUZHMualK5-ihvePR6o4QmoZVOBGicKvmkL_KB9IQYtxVqm3P_vpZ2HnFkoRfar4_uJOjqC8OCo5A/non_transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/audio-space/master_playlist.m3u8

Expected behavior use the master url for the processes(useful for ended spaces)

Output If applicable, add the output of the command to help explain your problem.

Traceback (most recent call last):
  File "c:\users\test01\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\test01\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\test01\AppData\Local\Programs\Python\Python39\Scripts\twspace_dl.exe\__main__.py", line 7, in <module>
  File "c:\users\test01\appdata\local\programs\python\python39\lib\site-packages\twspace_dl\__main__.py", line 116, in main
    args.func(args)
  File "c:\users\test01\appdata\local\programs\python\python39\lib\site-packages\twspace_dl\__main__.py", line 176, in twspace
    twspace_dl.master_url = args.from_master_url
UnboundLocalError: local variable 'twspace_dl' referenced before assignment

Desktop (please complete the following information):

• OS: Windows
• Version 2022.1.13.0
• Installation method pip

Additional context Same problem in twspace_dl -d DYN_URL

Hi, I have worked on these download jobs before. I would suggest using this one instead of user_tweet https://twitter.com/i/api/fleets/v1/avatar_content?user_ids={userIds}&only_spaces=true as I have met some people does not share the spaces to tweet. This method allows multi-user.

The only problem is it may be connection aborted.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/Command/spaces2.py", line 85, in <module>
    lives = fleetAvatar(idList, authHeaders)
  File "/home/Command/spaces2.py", line 39, in fleetAvatar
    rep = s.get(f'https://twitter.com/i/api/fleets/v1/avatar_content?user_ids={userIds}&only_spaces=true', headers=headers)
  File "/home/.local/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/.local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))```

Is your feature request related to a problem? Please describe. About login / cookie

Describe the solution you'd like Check the cookie is expired, and if the cookie is not expired, not login

Additional context

twspace_dl.py -U https://twitter.com/[user] --input-cookie-file cookie.txt login -u [username] -p [password] -o cookie.txt

When I run script like this, twspace-dl only gets a cookie, and not download twitter space. As I reffered #27, we need to check cookie file first, and it exists and not expired, skip login and trying to get and download twitter space.

This method will be better when run as Docker or Service (We can run as only one commands).

Hi, I am using this script on Ubuntu 20.04.2.

I have installed the script from PyPI, but the script returns the following error when I try to download from a live twitter spaces (direct link) or user URL:

Is there anything I can do to fix it? Seems like I am the only one facing this issue so far. I am not sure is it due to some dependencies or other factors.

Also, when downloading using -U, some users are not being shown as live even thought the Spaces is still live. I made sure the user selected is the Host of the spaces.

Hi,

I'm totally noob using this kind of softwares and I would like to request a step by step user guide for to download a Twitter Space.

Kind regards,

Closes: #2

When executing twspace_dl -i <url> with -l or --log, log file (.twspace-dl.%Y-%m-%d_%H-%M-%S_%s.log) will be created in pwd.

twspace_dl -i "https://twitter.com/i/spaces/1LyGBojergoJN" -l

Describe the bug Unable to get guest token after multiple tries.

To Reproduce twspace_dl -i

Expected behavior Grab guest token and start download of the twitter space

Output If applicable, add the output of the command to help explain your problem.

Traceback (most recent call last):er:4
  File "C:\Users\Xanek\AppData\Local\Temp\ON66CE~1\__main__.py", line 142, in main
  File "C:\Users\Xanek\AppData\Local\Temp\ON66CE~1\twspace_dl\TwspaceDL.py", line 235, in download
  File "C:\Users\Xanek\AppData\Local\Temp\ON66CE~1\functools.py", line 969, in __get__
  File "C:\Users\Xanek\AppData\Local\Temp\ON66CE~1\twspace_dl\TwspaceDL.py", line 138, in metadata
  File "C:\Users\Xanek\AppData\Local\Temp\ON66CE~1\twspace_dl\TwspaceDL.py", line 108, in guest_token
RuntimeError: No guest token found after five retry

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Xanek\AppData\Local\Temp\ON66CE~1\__main__.py", line 151, in <module>
  File "C:\Users\Xanek\AppData\Local\Temp\ON66CE~1\__main__.py", line 146, in main
AttributeError: 'TwspaceDL' object has no attribute '_tmpdir'

Desktop (please complete the following information):

• OS: [e.g. Windows] Windows 10
• Version [e.g. 2021.12.10.1] 19043.1415
• Installation method (binary, pip, or other) .exe

Additional context Add any other context about the problem here.

Describe the bug On a Twitter space that recently ended, saw this error for some minutes. Retrying the same download again a bit later worked fine. Maybe just a transient Twitter issue with the data they provided for a recently ended space, and not something twspace-dl needs to worry about.

To Reproduce Tried to download a recently ended space (ended a few minutes prior):

$ twspace_dl -m -i https://twitter.com/i/spaces/1DXxyDqgdbNJM?s=20

Expected behavior Download the space successfully

Output

./tmppau4qu6q/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8: Invalid data found when processing input
Error: ffmpeg -protocol_whitelist file,https,tls,tcp -y -stats -v warning -i ./tmppau4qu6q/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8 -c copy -metadata title=MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org -metadata artist=Mriya Report -metadata episode_id=1DXxyDqgdbNJM ./tmppau4qu6q/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m4a
Retry with -v to see more details
[[email protected] ~/Desktop/2022-08-10_1]$  twspace_dl -m -v -i https://twitter.com/i/spaces/1DXxyDqgdbNJM?s=20
2022-08-10 11:55:37,119 [DEBUG] Starting new HTTPS connection (1): api.twitter.com:443
2022-08-10 11:55:37,208 [DEBUG] https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 63
2022-08-10 11:55:37,213 [DEBUG] Starting new HTTPS connection (1): twitter.com:443
2022-08-10 11:55:37,473 [DEBUG] https://twitter.com:443 "GET /i/api/graphql/jyQ0_DEMZHeoluCgHJ-U5Q/AudioSpaceById?variables=%7B%22id%22%3A%221DXxyDqgdbNJM%22%2C%22isMetatagsQuery%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withUserResults%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withReplays%22%3Atrue%2C%22withScheduledSpaces%22%3Atrue%7D HTTP/1.1" 200 8031
2022-08-10 11:55:37,476 [DEBUG] Media Key: 28_1557383083459952641
2022-08-10 11:55:37,480 [DEBUG] Starting new HTTPS connection (1): cdn.syndication.twimg.com:443
2022-08-10 11:55:37,549 [DEBUG] https://cdn.syndication.twimg.com:443 "GET /widgets/followbutton/info.json?screen_names=MriyaReport HTTP/1.1" 200 156
2022-08-10 11:55:37,559 [DEBUG] Starting new HTTPS connection (1): twitter.com:443
2022-08-10 11:55:37,769 [DEBUG] https://twitter.com:443 "GET /i/api/1.1/live_video_stream/status/28_1557383083459952641 HTTP/1.1" 200 1498
2022-08-10 11:55:37,775 [DEBUG] Starting new HTTPS connection (1): prod-fastly-eu-central-1.video.pscp.tv:443
2022-08-10 11:55:38,020 [DEBUG] https://prod-fastly-eu-central-1.video.pscp.tv:443 "GET /Transcoding/v1/hls/zEg3Tz98sqlLU67tjex73hlyWdJsIfdcR11UO7Dj41CTBXT_Wsma_HszidusGs0qbpeGcwyJvswODSeDyJcmTQ/non_transcode/eu-central-1/periscope-replay-direct-prod-eu-central-1-public/audio-space/master_playlist.m3u8 HTTP/1.1" 200 550
2022-08-10 11:55:38,025 [DEBUG] Starting new HTTPS connection (1): prod-fastly-eu-central-1.video.pscp.tv:443
2022-08-10 11:55:38,282 [DEBUG] https://prod-fastly-eu-central-1.video.pscp.tv:443 "GET /Transcoding/v1/hls/zEg3Tz98sqlLU67tjex73hlyWdJsIfdcR11UO7Dj41CTBXT_Wsma_HszidusGs0qbpeGcwyJvswODSeDyJcmTQ/transcode/eu-central-1/periscope-replay-direct-prod-eu-central-1-public/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsInZlcnNpb24iOiIyIn0.eyJIZWlnaHQiOjgsIkticHMiOjM1MCwiVHJhbnNjb2RlQXVkaW8iOnRydWUsIldpZHRoIjo4fQ.j9A1sJbguxIY34C17U6iN7yTx8JM_ZNYJhfxaSmwGKQ/audio-space/playlist_16786592591867834374.m3u8 HTTP/1.1" 404 10
2022-08-10 11:55:38,285 [DEBUG] ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8 written to disk
2022-08-10 11:55:38,285 [DEBUG] Command for the old part: ffmpeg -protocol_whitelist file,https,tls,tcp -y -stats -v warning -i ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8 -c copy -metadata title=MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org -metadata artist=Mriya Report -metadata episode_id=1DXxyDqgdbNJM ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m4a
./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8: Invalid data found when processing input
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/twspace_dl/twspace_dl.py", line 171, in download
    subprocess.run(cmd_old, check=True)
  File "/usr/local/Cellar/[email protected]/3.9.13_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ffmpeg', '-protocol_whitelist', 'file,https,tls,tcp', '-y', '-stats', '-v', 'warning', '-i', './tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8', '-c', 'copy', '-metadata', 'title=MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org', '-metadata', 'artist=Mriya Report', '-metadata', 'episode_id=1DXxyDqgdbNJM', './tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m4a']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/twspace_dl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/twspace_dl/__main__.py", line 231, in main
    args.func(args)
  File "/usr/local/lib/python3.9/site-packages/twspace_dl/__main__.py", line 123, in space
    twspace_dl.download()
  File "/usr/local/lib/python3.9/site-packages/twspace_dl/twspace_dl.py", line 173, in download
    raise RuntimeError(" ".join(err.cmd)) from err
RuntimeError: ffmpeg -protocol_whitelist file,https,tls,tcp -y -stats -v warning -i ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m3u8 -c copy -metadata title=MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org -metadata artist=Mriya Report -metadata episode_id=1DXxyDqgdbNJM ./tmp0d075lpp/(Mriya Report)MriyaReport - Russian Invasion of 🇺🇦 Crimea Actions ⛑️ via MriyaAid.org-1DXxyDqgdbNJM.m4a

Desktop (please complete the following information):

• OS: MacOS 11.6.8 (20G730) (x86)
• Version: 2022.6.6.1
• Installation method: pip (python3)

Additional context Workaround: wait some time and retry the same download. Appears to be a temporary Twitter issue with the data Twitter returns shortly after a space ends. Wait a few minutes or more and try the same download again, it should work.

Is your feature request related to a problem? Please describe. People can change their Twitter handle/username whenever, I'd prefer to use the immutable user ID (for example 44196397)

Describe the solution you'd like Add %(creator_id)s

Describe the bug When downloading ongoing Space, the final file would have duplicate audio segments at the place of merging.

If I understand the code correctly, twspace-dl works by downloading existing segments using master_playlist.m3u8, then downloading new parts using the dyn url, then merge them together at the end.

The issue here is that no attempt is made to make sure the two don't have overlapping. And in my observation, they do.

cut.m4a.zip

In this example (cut the very first 30s of a long space), one or two segments repeat at 17.7s and 23.7s.

This issue only happens with running Space so it's kinda hard to give a easy STR.

Desktop (please complete the following information):

• OS: [e.g. Windows] Windows
• Version [e.g. 2021.12.10.1]
• Installation method (binary, pip, or other) pip

Describe the bug After fixing #71 in commit 7fad013, a new bug was introduced that the files in the temporary directory will be removed but not the directory itself if --keep-files was NOT used.

To Reproduce twspace_dl --input-cookie-file twitter.txt -suU "https://twitter.com/username" -m -v

Expected behavior Temporary directories and their content got removed after running the command when --keep-files was not used.

Output

Desktop (please complete the following information):

• OS: Ubuntu Linux 22.04.1 LTS
• twspace-dl Version: 6c57161
• Installation method: pip from GitHub repo

Simplest way to fix #63: consider download finished if there is existing file in working directory that has the same name as one that would be created.

In situation when space ended but still available for replay, running twspace_dl -U more than once will result in .m4a file being downloaded again and replacing already existing one. If systemd unit with RestartSec=15 is used, new download will be started as soon as old one finishes the entire time space stays available.

Checking if file with given name already exists before starting download would prevent this from happening. yt-dlp also has --download-archive option, which maintains list of id of successful downloads and skips over them if called twice.

Describe the bug A clear and concise description of what the bug is.

To Reproduce python3 ./twspace_dl/__main__.py -i https://twitter.com/i/spaces/1MYxNnkdkPPxw -v

Expected behavior begin to download the space audio.

Output If applicable, add the output of the command to help explain your problem.

2022-06-18 02:27:50,086 [DEBUG] Starting new HTTPS connection (1): api.twitter.com:443
2022-06-18 02:27:50,718 [DEBUG] https://api.twitter.com:443 "POST /1.1/guest/activate.json HTTP/1.1" 200 62
2022-06-18 02:27:50,727 [DEBUG] Starting new HTTPS connection (1): twitter.com:443
2022-06-18 02:27:51,452 [DEBUG] https://twitter.com:443 "POST /i/api/1.1/onboarding/task.json?flow_name=login HTTP/1.1" 200 246
2022-06-18 02:27:51,665 [DEBUG] https://twitter.com:443 "POST /i/api/1.1/onboarding/task.json HTTP/1.1" 200 786
2022-06-18 02:27:51,856 [DEBUG] https://twitter.com:443 "POST /i/api/1.1/onboarding/task.json HTTP/1.1" 400 76
Traceback (most recent call last):
  File "./twspace_dl/__main__.py", line 726, in login
    self.flow_token = request_flow.json()["flow_token"]
KeyError: 'flow_token'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./twspace_dl/__main__.py", line 1009, in <module>
    main()
  File "./twspace_dl/__main__.py", line 1004, in main
    args.func(args)
  File "./twspace_dl/__main__.py", line 849, in space
    auth_token = Login(args.username, args.password,
  File "./twspace_dl/__main__.py", line 728, in login
    raise RuntimeError("Error identifying user:",
RuntimeError: ('Error identifying user:', {'errors': [{'code': 366, 'message': 'Missing data.'}]})

Desktop (please complete the following information):

• mbp Monsterey 12.3 python3.8.9 64bit
• Version [e.g. 2022.6.10]
• Installation method (pip)

Additional context when I use poetry or pip:

poetry run twspace-dl -i https://twitter.com/i/spaces/1MYxNnkdkPPxw

  FileNotFoundError

  [Errno 2] No such file or directory: b'/Users/diven/.rvm/bin/twspace-dl'

twspace-dl is not in command, so I run it in source code lib

Is your feature request related to a problem? Please describe. This is more like a nice to have. But I think it'd be great if we have an option to automatically upload downloaded content to say an s3 bucket. This is useful when others use twspace-dl to build something like a webapp or just want things organized nicely in S3.

Describe the solution you'd like

1. Add an option to take a S3 bucket
2. A module to use S3 SDK to upload to the S3 bucket with credentials from .env

Describe alternatives you've considered Upload manually

Happy to help if the approach can be aligned :)