Categories
Mastering Development

Flutter/Firebase data pagination – again

I have a Firebase query going on into a 10k database. I don’t want the query to take ages, so I get 50 records at a time, process them for what I want, and store the final record in each search as the reference point to start the next search from. So essentially something like: […]

Categories
Database Development

Is it good database design to store large hashes in your tables?

I have to produce a lot of graphs using a lot of historical data (1000+ data points for each graph), and depending on some pre-defined options, I need to display that specific data in a graph. Let’s say I have 4 possible combinations of options X and Y, (X,Y),(X,X),(Y,X),(Y,Y) If each of these combinations takes […]

Categories
Database Development

LOAD DATA INFILE Performance Degrading

I have very little database experience, but I’ve inherited a database for a production system that has major problems that I am attempting to fix. The system receives a continuous stream of data from internet of things devices that my company uses to monitor manufacturing equipment. Each record is 180 bytes, and I receive about […]

Categories
Database Development

LOAD DATA INFILE Performance Degrading

I have very little database experience, but I’ve inherited a database for a production system that has major problems that I am attempting to fix. The system receives a continuous stream of data from internet of things devices that my company uses to monitor manufacturing equipment. Each record is 180 bytes, and I receive about […]

Categories
Development User Help

The disadvantages of having too many files in single directory

I have two options: Either: To have a database with 100,000 records, Or – To have these records as separate (and very small) text files in the single directory. What are the disadvantages of the second approach, except the copying will take a long time? I ask about the disadvantages from the performance point of […]

Categories
Development

Why is MEMBER OF() faster than JSON_CONTAINS()?

I’m playing with MySQL 8’s new JSON capabilities, in particular multi-value indexes. I noticed that there are 2 ways to check if a JSON array contains a given value: the MEMBER OF() operator, and the JSON_CONTAINS() function. They both return the same set of results for every query I make, but surprisingly, MEMBER OF seems […]

Categories
Development

Twilio: Is there a way to export all opt outs?

I want to export everyone who has reply stop (opt out) for my messages. I see that my options are API. However, through the API I can see that I get everything in pages of 50 messages, so I have to go and do a lot of code to go page by page, message by […]

Categories
Development Heroku Ruby

Making ruby garbage collection kick in and run on heroku

I have a web service in a Ruby on Rails app hosted on heroku. When this service runs, a lot of memory gets used. I call the service in batches, so after each batch, all that memory is no longer needed. However, it goes above the heroku quota, and starts firing R14 errors. Monitoring the […]