Feature innertube client fix - #219
Conversation
1da876d to
5276825
Compare
|
Hi, it's been a while without any feedback. For
|
|
Great work on this; still need to finish reviewing. Should have
something back to you this weekend
|
|
Thanks for the feedback. It gives a peace of mind for me. No need to rush. I am open for any improvement suggestions to this pull request. |
| print('Unable to access ' + player_file) | ||
|
|
||
| signature_timestamp = None | ||
| signature_timestamp_cache = settings.data_dir + '/sts_' + player_version + 'txt' |
There was a problem hiding this comment.
Recommend using os.path.join here. Also, .txt, not txt
There was a problem hiding this comment.
Sorry for late reply.
I've used os.path.join in place of string concatenation in this file and similar places.
| response_dict = json.loads(response) | ||
| if settings.use_visitor_data: | ||
| if not settings.use_po_token: | ||
| if response_dict['responseContext'].get('visitorData'): |
There was a problem hiding this comment.
We shouldn't assume 'responseContext' will be present - otherwise it will raise an exception when youtube changes something.
There was a problem hiding this comment.
I've put this inside try ... except block, with specific KeyError exception message.
| if settings.use_visitor_data: | ||
| if not settings.use_po_token: | ||
| if response_dict['responseContext'].get('visitorData'): | ||
| if not os.path.exists(visitor_data_file): |
There was a problem hiding this comment.
Not sure how the visitor system works - but do we want to refresh this file ever? Maybe YouTube issues an updated token for example and marks the old one as invalid?
There was a problem hiding this comment.
I added os.path.getmtime check to make sure that the visitorData.txt file is less than 86400 seconds old before using its content. Otherwise, the visitor data file will be deleted and replaced with new one.
| else: | ||
| if os.path.exists(visitor_data_file): | ||
| print('Removing visitor_data file') | ||
| os.remove(visitor_data_file) |
There was a problem hiding this comment.
For anonymity's sake - do we want to consider refreshing the visitor data every day? Again, not really sure what constraints go into it
There was a problem hiding this comment.
Done, as I mentioned above.
| with open(visitor_data_file, "r") as file: | ||
| visitor_data = file.read() | ||
| file.close() | ||
| except: |
There was a problem hiding this comment.
Use except Exception, otherwise you'll catch KeyboardInterrupt and SystemExit: https://stackoverflow.com/questions/54948548/what-is-wrong-with-using-a-bare-except
There was a problem hiding this comment.
I added except OSError to notify if there is a file access error which prevents access to the visitor data file.
|
|
||
| def extract_nsig_func(base_js): | ||
| for i, member in enumerate(NSIG_FUNCTION_ARRAYS): | ||
| func_array_re = regex.compile(member.replace('$', '\\$')) |
There was a problem hiding this comment.
I recommend against this; I would just put the three \ escapes you need into your regex instead of modifying it at runtime
There was a problem hiding this comment.
I only tried doing what inv_sig_helper does, copying exactly the same regex pattern with runtime replacement of escaping dollar sign which only done once for as long as the extracted nsig_func_{player_version}.js file exists.
The resulting n_sig_code is cached as data/nsig_func_{player_version}.js and loaded as info['nsig_func'] = { player_version: js_nsig_decrypt_code } during runtime of the youtube-local` session.
So the n_sig_function extraction is only done once and the subsequent access to it is either loaded directly from the info['nsig_func'] dict or loaded from nsig_func_{player_version}.js file if the file is already exists.
There was a problem hiding this comment.
Done, as requested by the project author.
| func_body_re = [] | ||
| for i, member in enumerate(NSIG_FUNCTION_ENDINGS): | ||
| func_body_re_item = '' | ||
| func_body_re_item += func_context.group(1) |
There was a problem hiding this comment.
You need to do a re.escape() on this before appending it to your regexes
| print('jscode len is: ' + str(len(jscode))) | ||
| dukpy_session = dukpy.JSInterpreter() | ||
| # Loading the function into dukpy session | ||
| dukpy_session.evaljs(jscode) | ||
| print('n_sig = ' + n_sig) | ||
| #n_sig_result = dukpy_session.evaljs('decrypt_nsig("' + n_sig + '")') | ||
| n_sig_result = dukpy_session.evaljs("decrypt_nsig(dukpy['n'])", n=n_sig) | ||
| print('n_sig_result = ' + n_sig_result) | ||
| return n_sig_result |
There was a problem hiding this comment.
Have we verified that dukpy has limited execution privileges? For instance, can javascript code executed with Dukpy make network requests or open files? If so it would be a massive security hole
Also recommend removing these debugging print statmenets when you're done
There was a problem hiding this comment.
dukpy is just a python wrapper for duktape js engine.
The pypi package of dukpy has dukpy-install command which is able to download npm packages from the internet.
Unless told to do so, dukpy module doesn't access the internet for as far as I know. The nsig_func doesn't need access to the internet during runtime, which I have verified doing manual n_sig decryption using various python js bindings.
I also consider dukpy as just-work lightweight js engine for python, since it has wheels for arm64 on pypi and armhf on piwheels.org so if anyone runs this on their single board computers, they will hopefully meet no problems during runtime.
| cachetools>=4.0.0 | ||
| stem>=1.8.0 | ||
| fake-useragent>=1.5.1 | ||
| flpc>=0.2.5 |
There was a problem hiding this comment.
Did you do any performance testing that suggested the need for this? Is there a noticeable speedup? Would rather avoid dependencies if possible
There was a problem hiding this comment.
With no break statement, the re module will hang for some time. flpc will not hang in finding the specified regex, even without break statement in the for loop.
I added the break statement in the for loop so the regex engine will be freed from work (i.e. testing another regex pattern) after a match is found, which mitigates hanging on the built-in re module.
I've removed flpc from the requirements to use the built-in re module as you wish, with very small or no performance degradation during my extended testing.
|
Any chance you could make a version of this similar to the youtube local main branch that uses Python 3.6 or earlier so it can run on Windows 7? |
I actually tried several times to also build packages for So I reverted my github action recipe to only build for python 3.11 on Windows. |
f50cae3 to
c7e1cac
Compare
|
Update for today: Currently experiencing 403 errors with Full video can only be played on |
Update for today: It seems that the cause of 1 minute playable stream is
Adding Btw, Any improvement suggestion is appreciated. |
217bb5a to
bd210e4
Compare
1f39c3a to
321d15e
Compare
|
Currently encountering issue with player |
|
Hi, for those who are interested, I have created my own simple YT web player based on Quart and yt_dlp. Check it out. |
|
Haven't had bandwidth to engage with this repo in awhile, sorry. Is this PR ready to go or still needed, do you want me to rereview it? I recall the hotfix you provided awhile back fixed most videos (except age restricted and a couple copyrighted videos maybe); does this fix address anything else currently? |
|
This branch is working for my needs, i.e. a self hosted instance, and the accumulated changes has gotten too much to be reviewed. The issue I encountered myself is there is probability of race condition when YT updated the player version when there is older cached |
2f3f058 to
e3f450e
Compare
|
Currently unable to extract n-signature for player The issue also happens with third party yt frontend out there. |
|
Currently, the hardcoded player version causes The robust solution for js code extraction is using AST based parser, which acts like a I've looked at the result of js code extraction by It's time to get For anyone out there experiencing |
|
Ok, finally got ast-based signature solver working via One of |
|
Got hit by another Still investigating the possibility to integrate |
|
Ok, finally got It will fetch |
3783319 to
6f807f7
Compare
Update: Tested to work with |
|
When will a new release be made? |
A new release has been built, tagged Feel free to try it. |
6f807f7 to
20fa801
Compare
|
Looks like there are quite large changes in master branch after 2.8.14 that makes this branch requires more often rebase works. |
|
A full rework of this feature is on the way. Will force-push later if I think it's good enough. @user234683 : by the way, how does a working js innertube client sound? Nice to have or unnecessary complicated? |
Use mweb client definition from yt-dlp project.
Allow users to choose between non-js client and js client.
To use js client, bgutil-ytdlp-pot-provider server has to bind on localhost:4416.
Add a new submodule to provide functions related to js signature decryption. One of [ 'deno', 'node', 'bun' ] runtime is required to be present on $PATH to decrypt js signatures so the js player client is able to get working stream url, in addition to bgutil-ytdlp-pot-provider server on localhost:4416
Use settings.player_client to determine which client to use for player api request. The user is responsible to set up a js runtime and pot server for the js player client to work.
Add yt-dlp-ejs>=0.8.0
Add yt-dlp-ejs>=0.8.0, which requires python>=3.10, potentially breaking oldwin.
Add yt-dlp-ejs>=0.8.0
d4dd6bd to
3373f40
Compare
I think it is going to be necessary ultimately. I don't see this visionos workaround lasting very long. General structure of what you have so far looks promising (haven't reviewed in depth). Will review once you let me know it's ready |
Catch yt-dlp-ejs `ImportError` exception and show a fallback warning when js decryption is not possible.
Fall back to non-js client when js decryption is not possible.
|
Thanks for expressing your impression of I added some minor changes so the failure of With my last two commits, the app will fall back to non |
5 participants
The major change is the addition of
mwebinnertube client, which includes some refactoring of howbase_jsis handled and a new trick to decryptnsignature by extracting the relevant decryption code frombase_jsfile,using technique similar to. Update: the signature decryption is based oniv-org/inv_sig_helperyt-dlp-ejs.This pull request also introduce three dependencies:, andfake-useragentto simplify user agent header creation of mobile and desktop browser,flpcto parse nsig decryption regexdukpyjs_runtimes1 to execute the extracted nsig decryption code.Also add the ability to parse
visitorDatain the YT Api response and specifying ownvisitorDataandpoTokenpair using properly formatted json in thedata/po_token_cache.txtfile.Also several more fixes for
androidandiosclient and makeinnertubeclient selectable.Also several changes in
settings.py, notably to allow reloading oftv_embeddedclient in case of missing player urls and showingDownloadplaceholder viause_video_downloadoption, which credits~heckyel/yt-local.This will hopefully fix #218. Update: it has been fixed for a long time.Update:No more prebuilt release forfake-useragenthas been removed from therequirements.txtsince the module doesn't work with python 3.7 (oldwin) and 3.8 (latest version supporting win7).oldwin.Also added a workaround to get latest
INNERTUBE_CONTEXTfor web, mweb, and tv client viaytcfg.Prebuilt releases for Windows (both 64 bit
and 32 bit oldwin2) can be found on my releases page.Footnotes
One of
[ 'deno', 'bun', 'node'], usingyt-dlp-ejspackage. Make sure the js runtime is available in$PATH(i.e. bring your own runtime) before running theserver:application, otherwise the web innertube clients won't work. ↩No more
oldwinbuild, because the integration ofyt-dlp-ejsrequirespython>=3.10↩