Read avro files in pyspark with PyCharm

I’m quite new to spark, I’ve imported pyspark library to pycharm venv and write below code: # Imports from pyspark.sql import SparkSession # Create SparkSession spark = SparkSession.builder \ .appName(‘DataFrame’) \ .master(‘local[*]’) \ .getOrCreate() spark.conf.set(“spark.sql.shuffle.partitions”, 5) path = “file_path” df = spark.read.format(“avro”).load(path) , everything seems to be okay but when I want to read avro…

Details

Nixos configuration use pinned nixpkgs for specific package

So my nixos-rebuild fails to build due to the recent error of: https://github.com/NixOS/nixpkgs/issues/74916 I thought I’d give a go at trying to fix this, by referencing a previous commit’s package I added the following in my config (under one of the let statements): nixpkgsMyStable = import (builtins.fetchTarball { url = https://github.com/NixOS/nixpkgs/archive/775fb69ed73e7cf6b7d3dd9853a60f40e8efc340.tar.gz; sha256 = “1w068b0ydw4c26mcjiwlzdfqcdk3rrwmfx4hxzgfhfwcz2nmh3if”; })…

Details

DCHP server on virtual machine in HNVv1 virtual network

I’ve got a virtual machine with Windows Server 2016 Datacenter, Hyper-V nested virtualization on the host operating system switched on for this virtual machine and Hyper-V role installed on it. PS C:\> Set-VMProcessor -VMName hvm -ExposeVirtualizationExtensions $true PS C:\> Install-WindowsFeature -Name Hyper-V -IncludeManagementTools -Restart Two more virtual machines (nested) with Debian Linux deployed on this…

Details