Categories
Mastering Development

SLURM task fails when creating an instance of the Dask LocalCluster in an HPC cluster

I’m queuing a task with the command sbatch and the next configuration: #SBATCH –job-name=dask-test #SBATCH –ntasks=1 #SBATCH –cpus-per-task=10 #SBATCH –mem=80G #SBATCH –time=00:30:00 #SBATCH –tmp=10G #SBATCH –partition=normal #SBATCH –qos=normal python ./dask-test.py The python script is, more or less, as follows: import pandas as pd import dask.dataframe as dd import numpy as np from dask.distributed import Client, […]

Categories
Mastering Development

Combine data from two csv files and output into categories

I have two csv files: one with files/paths that have been altered and one with files/paths that have been deleted. I am trying to combine the two to make it easier to see what has been added, altered, or deleted. For example in the csv with altered lines: Hello_World.py,/Users/name/DropBox/Other Test Essay.docx,/Users/name/DropBox/Other Test/XXX NEW Project.docx,/Users/name/DropBox/Other Test/XXX […]

Categories
Mastering Development

Using csv how can u split an row to create independent data in variables? (My attempt bellow)Can u help improve it, thanks in advance.Python

def Start(): with open(‘Songs.csv’) as Data #Call file Read = csv.reader(Data,delimiter=’:’, quotechar=’|’)#return tuple of songs and artists for row in Read : X = random.choice(row) Store = (‘,’.join(X)) A = re.compile(‘,’) for s in finditer(Store): POS = s.start()#positional arg for slicing global Song Song = X[POS:] global Artist Artist = X[:POS] print(Song)

Categories
Mastering Development

XML parsing with a Class call

I parsed an xml file with xml.etree python module. Its Working well, but now I try to call this code as a module/Class from a main program. I would like to send the xml tree and a filename for the csv writing to the Class. My dummy file to call the file with the Class: […]

Categories
Mastering Development

Finding a slope of a given point in a data set using python

I have a data in a txt file that is comprised of two columns that I can retrieve and put as two arrays using numpy (X,Y). os.getcwd() os.chdir(‘C:\\python’) xvalue = np.loadtxt("test.txt", delimiter= " ")[:, 0] yvalue = np.loadtxt("test.txt", delimiter= " ")[:, 1] x = np.array(xvalue) y = np.array(yvalue) zip(x,y) Assuming that the data is something […]

Categories
Mastering Development

Python: Iterating Large Files

I am a beginner and will appreciate any alternatives to handle my problem. Simply put, I have two files, containing one vector each. Aim is to subtract all the elements of file 2 from file 1; for all possible combinations. Everything is fine for small vectors, everything is fine, but the processing time is huge […]

Categories
Linux Mastering Development

How to write grid in csv file in python

I have a list of tuples. Each tuple contain 2 values, together with the results of an operation between the two values. Here is an example: my_list = [(1,1,1.0), (1,2,0.8), (1,3,0.3), (2,1,0.8), (2,2,1.0), (2,3,0.5), (3,1,0.3), (3,2,0.5), (3,3,1.0)] I need to store this value in a csv file so that they look like this: 0 1 […]

Categories
Mastering Development

TypeError: Mismatch between array dtype (‘

I am using np.savetxt for the first time, and I am trying to save two variables (a string and a float) in a file named “trial.csv” as follows: import numpy as np RT = 2.76197329736740 key_name = ‘space’ print(RT,key_name) # Save data in a CSV file named subj_data_file np.savetxt(“trial.csv”, (RT,key_name), delimiter=’,’, header=”RTs,Key_Name”) I got the […]

Categories
Mastering Development

Multiple Errors During HDF5 to CSV conversion

I have a huge h5 file wich I need to extract each data-set into a separate csv file. The schema is something like /Genotypes/GroupN/SubGroupN/calls with ‘N’ groups and ‘N’ sub-groups. I have created sample h5 file with same structure as main file and tested the codes which worked correctly but when i apply the code […]

Categories
Mastering Development

I would like to export DynamoDB Table to S3 bucket in CSV format using Python (Boto3)

This question has been asked earlier in the following link: How to write dynamodb scan data's in CSV and upload to s3 bucket using python? I have amended the code as advised in the comments. The code looks like as follows: import csv import boto3 import json dynamodb = boto3.resource(‘dynamodb’) db = dynamodb.Table(’employee_details’) def lambda_handler(event, […]