Technology Blog

This morning I had to troubleshoot a third party module (Amasty Out of Stock Notifications) that was not sending emails out properly. After wrongfully thinking the module was junk (we used it on M1 and it worked great, but had doubts about the m2 version), I jumped full blown dev into coding / debugging mode.

This lead me to this line:


if ($allProducts && $product->isSalable()) {


So I of course printed out the return values of these. $allProducts returned a collection but isSalable returned nothing. Not a zero and not a 1.

Looking into isSalable is just a very big fun mess that does a whole lot of things that I never really ever wanted to look under the hood about. Luckily, I realized it used a bunch of stuff form the indexer….

That’s when I headed over to my indexer status page in the admin panel and saw they were processing. Then I noticed they have been processing always. Every minute the cron fires off, they pick back up. That’s not right….

So I decided to run them via CLI


[staging@dev htdocs]$ php bin/magento indexer:reindex
Table status for design_config_dummy_cl is incorrect. Can`t fetch version id.
Table status for customer_dummy_cl is incorrect. Can`t fetch version id.
Table status for catalog_product_flat_cl is incorrect. Can`t fetch version id.
Table status for catalog_category_flat_cl is incorrect. Can`t fetch version id.
Table status for catalog_category_product_cl is incorrect. Can`t fetch version id.
Table status for catalog_product_category_cl is incorrect. Can`t fetch version id.
Table status for catalogrule_rule_cl is incorrect. Can`t fetch version id.
Table status for catalog_product_attribute_cl is incorrect. Can`t fetch version id.
Table status for cataloginventory_stock_cl is incorrect. Can`t fetch version id.
Table status for inventory_cl is incorrect. Can`t fetch version id.
Table status for catalogrule_product_cl is incorrect. Can`t fetch version id.
Table status for catalog_product_price_cl is incorrect. Can`t fetch version id.
Google Product Removal Feed index is locked by another reindex process. Skipping.
Google Product Feed index is locked by another reindex process. Skipping.
Table status for catalogsearch_fulltext_cl is incorrect. Can`t fetch version id.
Table status for amasty_feed_entity_cl is incorrect. Can`t fetch version id.
Table status for amasty_feed_product_cl is incorrect. Can`t fetch version id.


What the heck! Luckily, thanks to some stack overflow articles I found out that either the auto_increment value has been corrupted or mysql just needs to refresh some kind of table status cache it has. So the fix here was simply:


analyze table `design_config_dummy_cl`;
analyze table `customer_dummy_cl`;
analyze table `catalog_product_flat_cl`;
analyze table `catalog_category_flat_cl`;
analyze table `catalog_category_product_cl`;
analyze table `catalog_product_category_cl`;
analyze table `catalogrule_rule_cl`;
analyze table `catalog_product_attribute_cl`;
analyze table `cataloginventory_stock_cl`;
analyze table `inventory_cl`;
analyze table `catalog_product_price_cl`;
analyze table `catalogsearch_fulltext_cl`;
analyze table `amasty_feed_entity_cl`;
analyze table `amasty_feed_product_cl`;


ANalyze, causes that “internal cache” nonsense or whatever if it, so update and then our indexer ran just fine on one of our test servers.

On another test server, we were met with more errors:


Catalog Product Rule indexer process unknown error:

SQLSTATE[HY000]: General error: 1419 You do not have the SUPER privilege and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable), query was: CREATE TRIGGER trg_catalog_product_entity_after_insert AFTER INSERT ON catalog_product_entity FOR EACH ROW
BEGIN
INSERT IGNORE INTO `catalogrule_product_cl` (`entity_id`) VALUES (NEW.`entity_id`);
INSERT IGNORE INTO `scconnector_google_remove_cl` (`entity_id`) VALUES (NEW.`entity_id`);
INSERT IGNORE INTO `scconnector_google_feed_cl` (`entity_id`) VALUES (NEW.`entity_id`);
END

Product Price indexer process unknown error:

SQLSTATE[HY000]: General error: 1419 You do not have the SUPER privilege and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable), query was: CREATE TRIGGER trg_catalog_product_entity_after_insert AFTER INSERT ON catalog_product_entity FOR EACH ROW
BEGIN
INSERT IGNORE INTO `catalog_product_price_cl` (`entity_id`) VALUES (NEW.`entity_id`);
INSERT IGNORE INTO `scconnector_google_remove_cl` (`entity_id`) VALUES (NEW.`entity_id`);
INSERT IGNORE INTO `scconnector_google_feed_cl` (`entity_id`) VALUES (NEW.`entity_id`);
END

Luckily here, the fix was pretty easy too:

just log into mysql from cli, and run:


set global log_bin_trust_function_creators=1;


And then all should be well:


[staging@dev htdocs]$ php bin/magento indexer:reindex

Design Config Grid index has been rebuilt successfully in 00:00:00
Customer Grid index has been rebuilt successfully in 00:00:03
Product Flat Data index has been rebuilt successfully in 00:00:01
Category Flat Data index has been rebuilt successfully in 00:00:00
Category Products index has been rebuilt successfully in 00:00:00
Product Categories index has been rebuilt successfully in 00:00:00
Catalog Rule Product index has been rebuilt successfully in 00:00:00
Product EAV index has been rebuilt successfully in 00:00:00
Stock index has been rebuilt successfully in 00:00:00
Inventory index has been rebuilt successfully in 00:00:00
Catalog Product Rule index has been rebuilt successfully in 00:00:00
Product Price index has been rebuilt successfully in 00:00:04
Google Product Removal Feed index has been rebuilt successfully in 00:00:00
Google Product Feed index has been rebuilt successfully in 00:00:00
Catalog Search index has been rebuilt successfully in 00:00:00
Amasty Feed Rule index has been rebuilt successfully in 00:00:00
Amasty Feed Products index has been rebuilt successfully in 00:00:00


Lastly, for a healthy magento, You’ll want to make sure your indexer are set to run on a schedule:


php bin/magento indexer:set-mode schedule


So today I migrated a linux VM from vultr to an internal xcp-ng box.

We used clonezilla to store the image to a wasabi bucket. Then we used clonzeilla on the xcp-ng machine to bring it down.

We ran into an issue booting the server after the restore. We were getting Dracut errors and dropped into a dracut shell. Pretty scary stuff, and after a few hours of trying things the solution was really easy.


* From the GRUB menu, select the last one (the rescue)
* Then login, and do so as root or as a normal user and run su -
* run the following command:

dracut --regenerate-all --force


Then reboot the box.

I just wanted to write a review about my last interaction with ssls.com. Previously, we had always bought certain products from here and they were reliable, helpful, and always delivered.

Unfortunately, they have now turned into a deceitful company and have been engaging in possible fraud. That’s right – buyer be warned.

A few years ago, the SSL community changed the max age of ssl certificated from 5 years to 4 years, to 3 years, now down to 2 year max. This meant that average order values for companies like ssls.com plummeted by 50%. Unfortunately, their customer acquisition did not drop at all, it even possibly increased during this time span as more competitors came to light. Additionally, LetsEncrypt also came about offering free ssl certificates and eating into market share.

Now here comes the genius minds at ssls.com with a bridge to sell you (reference to the old, I have the Brooklyn Bridge to sell you scam). They start offering a 4 year certificate for you to buy. That’s right, the max is 2 years, but they begin selling a 4 year cert. When you dig into it, it’s 2 x 2yr certs. Cool, that’s fine I thought. Being in the industry I knew 4 years were not longer available and they found a neat work around to get bulk orders and they walk you through the proces sof what they are doing. Nothing shady there. Your getting a discount for buying 2 certs and you understand that after 2 years you have to activate the 2nd 2yr cert. Cool. Fair Game.

But then I guess sales weren’t that good. Because why buy 2 x 2yr certs when you really only need a cert for now and who knows what is going to happen in 2 yrs + from now. So I’m guessing sales weren’t too hot, because now we get onto the scam part. Now when you buy a 2yr certificate, they give you a 2 x 1 year certificates. They tell you to come back in one year to get your next year. Your not getting the 2 year certificate that you purchased.

That’s right, they make it a hassle to get your two years up front – they don’t even sell you it. It’s now 2 x 1yr certs you get. Which is not industry standard.

The problem here is that there is no mention of this on the product detail pages or anywhere else. They advertise and sell the product (a 2 yr certificate) as the same product that you buy anywhere else. They are a re-seller, so the same product is available elsewhere and when price comparing or shopping apples to apples, you really believe it is an apple your getting.

However, if you bought the 2 year certificate elsewhere, you’d be issued a 2 year certificate. You would not be issued a 1yr certificate and told to come back in 1yr to get a new certificate. Each time you come back is a hassle, you have to reactivate, possibly re-upload files, work with your hosting company, some hosting companies charge a fee for certificate installation and testing, etc.

So when your being told that your buying a 2 year certificate and then your being issues a 1 year certificate – that can only be explained as fraud.

If you want a 2yr certificate from them, your now stuck buying the fake 4 year certificate so you can get the 2 x 2yr certificates.

Imagine going to buy an Audi A8. imagine you price shop between two dealers and then you decide to go with the slightly cheaper dealership. Now imagine when you sign everything, and hand them cash, they bring you in the back and they give you an A4. They then tell you in a year to return this A4 and then they will give you another A4. That’s not what you bought! You bought an A8 – not an A4.

Plain and simple -this company has become a bunch of scam artist. Playing cheap tricks on their customers all in the name of a quick buck.

This has apparently started on December 11th (or so), 2019. I hope they see their scammy ways in 2020 and reverse ship. Otherwise previously loyal customers who have ordered 100+ certs will leave in droves.

We had someone do a rm -Rf /var instead of rm -Rf var inside a project folder. If your reading this, you probably know the pain.

The admin ctrl+c ed the rm after a few seconds/minutes when the command didn’t finish as quickly as he expected it to.

Lucky for us, we were able to run yum, and rpm -qa still worked.

If your in the same boat and


yum info


works, we have a shot of rebuilding the box.

The first step is to run


rpm -Va > /root/missing.txt


Once this completes you will want to print out just the missing stuff:


cat /root/missing.txt |grep missing


Next you will want to use a multi line select tool or sed or something to take each line that starts with missing and then populate rpm -q --whatprovides around it. For example:


rpm -q --whatprovides /var/account
rpm -q --whatprovides /var/lib/mailman
rpm -q --whatprovides /var/lib/mailman/archives
rpm -q --whatprovides /var/lib/mailman/archives/private
rpm -q --whatprovides /var/lib/mailman/archives/public
rpm -q --whatprovides /var/lib/mailman/data
rpm -q --whatprovides /var/lib/mailman/data/sitelist.cfg
rpm -q --whatprovides /var/lib/mailman/lists
rpm -q --whatprovides /var/lib/mailman/spam


You will now want to run these commands and take note of the rpm's returned.

For example:

[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/archives
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/archives/private
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/archives/public
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/data
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/data/sitelist.cfg
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/lists
mailman-2.1.12-26.el6_9.3.x86_64
[bessig@after ~]$ rpm -q --whatprovides /var/lib/mailman/spam
mailman-2.1.12-26.el6_9.3.x86_64


We can see we need to reinstall mailman-2.1.12-26.el6_9.3.x86_64.

So fire off:


yum reinstall mailman-2.1.12-26.el6_9.3.x86_64


Sometimes you may get a message from yum telling you it can no longer find that package. In this case you may need to run:


yum update mailman


In rare cases, you may need to find the rpm and manually reinstall it.

In our case, mysql was completely messed up, so we had to reinstall mysql, and then bring over the backup dumps and reimport them. Luckily for us, our most recent backup completed 4 minutes before this happened.

Once you think your done, rerun


rpm -Va > /root/missing.txt


and make sure this returns nothing:


cat /root/missing.txt |grep missing


Hope this helps. There's also a way to recover from a fully deleted /var directory (where the admin doesn't cancel the command).

I have done that before and will hope to do a write up in the future.