Categories
Linux Mastering Development

curl –data-binary $’ inside a shell script

I have tried everything to do a curl from within a shell script. This curl has a –data-binary $’ part which I just can not get my script to recreate. It will move the $ inside the single quote for some weird reason. Any ideas how to fix it? –data-binary ‘$ curl -s -H ‘Connection: […]

Categories
Linux Mastering Development

curl –data-binary $’ inside a shell script

I have tried everything to do a curl from within a shell script. This curl has a –data-binary $’ part which I just can not get my script to recreate. It will move the $ inside the single quote for some weird reason. Any ideas how to fix it? –data-binary ‘$ curl -s -H ‘Connection: […]

Categories
Linux Mastering Development

curl –data-binary $’ inside a shell script

I have tried everything to do a curl from within a shell script. This curl has a –data-binary $’ part which I just can not get my script to recreate. It will move the $ inside the single quote for some weird reason. Any ideas how to fix it? –data-binary ‘$ curl -s -H ‘Connection: […]

Categories
Linux Mastering Development

Performing a “complex” curl inside a shell script

I have tried everything to do a curl from within a shell script. This curl has a –data-binary $’ part which I just can not get my script to recreate. It will move the $ inside the single quote for some weird reason. Any ideas how to fix it? –data-binary ‘$ curl -s -H ‘Connection: […]

Categories
Mastering Development System & Network

nginx caches the same file with versions

I added default setup for cache on Nginx. On usual refresh I receive 304 but in the cache folder there are additional files with versions like 02a9299b4f7a308b588544c1a735cc1e.0000000002, 02a9299b4f7a308b588544c1a735cc1e.0000000004 and so on. I assumed that the same cache entry should be used for the same query. But it creates additional files even on usual request. I […]

Categories
Mastering Development

Loop through Dataframe column of URLs and parse out html tag

This shouldn’t be too hard, although I can’t figure it, i’m betting i’m making a dumb mistake. Here’s the code that works on an individual link and returns the zestimate (the req_headers variable prevents throwing a captcha): req_headers = { ‘accept’: ‘text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8’, ‘accept-encoding’: ‘gzip, deflate, br’, ‘accept-language’: ‘en-US,en;q=0.8’, ‘upgrade-insecure-requests’: ‘1’, ‘user-agent’: ‘Mozilla/5.0 (Windows NT 10.0; […]

Categories
Mastering Development

I am trying to scrape a wikipedia page in nodejs

I am trying to scrape a wikipedia page https://en.wikipedia.org/wiki/List_of_largest_companies_by_revenue. My target is to store all 50 rows in the table to csv file. But I am not able to do so. Please find the code attached. // Importing necessary modules const request = require(“request-promise”) const cheerio = require(“cheerio”) const fs = require(“fs”) const json2csv = […]

Categories
Mastering Development

the difference between a real chrome browser and chromedriver? [duplicate]

This website is able tell the difference between a real chrome browser and chromedriver. Does anybody know what is the difference between a real chrome browser and chromedriver? Thanks. https://www.impactaging.com/full/11/908 $ cat chrdvrget.py #!/usr/bin/env python3 import sys from selenium import webdriver options = webdriver.ChromeOptions() options.add_argument(‘–headless’) browser = webdriver.Chrome(‘chromedriver’, options=options) browser.get(sys.argv[1]) sys.stdout.write(browser.page_source) browser.close() $ ./chrdvrget.py https://www.impactaging.com/full/11/908 […]

Categories
Mastering Development

no-referrer-when-downgrade , needs clarification

My first request looks like below Request URL: http://hostname/login?url=https://someURL Request Method: GET Status Code: 302 Moved temporarily Remote Address: xxx.xx.xx.xx:8081 Referrer Policy: no-referrer-when-downgrade it’s Request header is pecified below. Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 Accept-Encoding: gzip, deflate Accept-Language: en-US,en;q=0.9 Cookie: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Host: hostname Proxy-Connection: keep-alive Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) […]

Categories
Mastering Development

Multi-select between parentheses

Let’s say I am looking to have the following highlighted in vim: That is, I would like to add the word “END” to the end of each json key, for example: “Accept-END” “Accept-Encoding-END” …etc What would be the best way to do this? If it were the same length I could do a visual control-select […]