Categories
Mastering Development System & Network

Migration of password hashes from glftpd 2.01 (PKCS5_PBKDF2_HMAC_SHA1) to proftpd + mod_sql_passwd

For migrating from the de-factor closed source glftpd 2.01 to proftpd I need to migrate password hashes of the user accounts from glftpd to proftpd. Reading about the topic I figured mod_sql_passwd should do the trick. I therefore set up my proftpd server like this: <global> SQLBackend mysql SQLAuthTypes Crypt SQLAuthenticate users groups SQLConnectInfo testdbuser@testdbhost […]

Categories
Development

SQL Bulk Insert

I have a csv file that is sent to me. I have put the file into Notepad++ which shows the ROWTERMINATOR as CRLF. This means that I use \r\n as my ROWTERMINATOR in SQL correct? This is the file format (CRLF at the end of each row): 2020/02/10 03:00:00,2020/02/11 02:59:59,2 101,1,5,1,7,10950,0,10950,0,1429 101,1,5,2,7,28878,5500,28878,0,0 101,1,5,3,4,9525,1200,9525,0,0 101,1,5,4,1,7686,0,7686,0,0 101,1,5,7,3,9094,1300,9094,0,0 […]

Categories
Development

Why does Spark run 5 jobs for a simple aggregation?

I use Spark in local mode from an IDE/eclipse. I can see Spark UI creating many jobs for a simple aggregation. Why? import org.apache.spark.sql.SparkSession trait SparkSessionWrapper { lazy val spark: SparkSession = { SparkSession .builder() .master(“local[2]”) .appName(“Spark Me”) .getOrCreate() } spark.sparkContext.setLogLevel(“WARN”) } The Spark application is as follows: object RowNumberCalc extends App with SparkSessionWrapper { […]

Categories
Development

Spark Creating REDUNDANT Jobs?

Local Mode Spark App is run from an IDE/eclipse and I see Spark UI creating **So Many Jobs ** for a simple Aggregation. I expect to see 1 Job with 2 stages what is happening here ?!! Is it for the SparkSession Settings ? Code trait => // import org.apache.spark.sql.SparkSession trait SparkSessionWrapper { lazy val […]

Categories
Development

Spark Creating REDUNDANT Jobs?

Local Mode Spark App is run from an IDE/eclipse and I see Spark UI creating **So Many Jobs ** for a simple Aggregation. I expect to see 1 Job with 2 stages what is happening here ?!! Is it for the SparkSession Settings ? Code trait => // import org.apache.spark.sql.SparkSession trait SparkSessionWrapper { lazy val […]

Categories
Development

Spark Creating REDUNDANT Jobs?

Local Mode Spark App is run from an IDE/eclipse and I see Spark UI creating **So Many Jobs ** for a simple Aggregation. I expect to see 1 Job with 2 stages what is happening here ?!! Is it for the SparkSession Settings ? Code trait => // import org.apache.spark.sql.SparkSession trait SparkSessionWrapper { lazy val […]

Categories
Development

Python – Concatenate error messages for specific columns having Null/NA

Notes: This is a data scrape from the link provided at the beginning of the code; not my data, just formatted by me I have a specific set of columns that cannot be blank I have an error message for each associated column I am trying to find ALL columns that are NA and assign […]