Kwynn's home
Your database is down / unreachable (MySQL). That is a possible / likely cause / solution.
This is one of the most deceptive error messages I've ever come across. So, to be clear, you get a deceptive "rebuild_theme" error when the database is down.
I show the whole error message / stack trace further below, under "(Original) Part 2."
Unsurprisingly, over the course of 2017 Drupal replaced database.inc and thus my changes were erased. Yes, yes, that is why one doesn't change the core. I understood that the first time, but at least my problem was solved for months, and I had the option to redo my changes. However, now I've figured out how to solve this problem without changing Drupal core / database.inc. (Database.inc may not be technically the "core," but whatever.)
Once my early 2017 changes were erased, my problem returned: specifically, I'm connecting Drupal code hosted on AWS to a Drupal database on Pantheon. Even with my 160s timeout, the database connection fails 10% of the time.
After perhaps an hour of tracing the execution path with Netbeans and XDebug, I discovered that 1. the PDO timeout default is 2 or 3 seconds, and 2. you can add a timeout to your connection array:
In /[Drupal root]/sites/default/settings.php : $databases = array ( 'default' => array ( 'default' => array ( 'database' => 'db1', 'username' => 'user1', 'password' => 'pwd', 'host' => 'localhost', 'port' => '3306', 'driver' => 'mysql', 'pdo' => array(PDO::ATTR_TIMEOUT => 160), ), ), );
Where 160 is 160s or 160 seconds. 160 seconds is probably pointlessly too long. I picked it for historical reasons. You may want to figure out what's more reasonable. I would take a guess at 20 - 30 seconds. I'm running a batch job in the middle of the night, though, so I don't care how long it waits.
Setting a timeout probably solves connection failure part of the time, but I'm still having a 10% failure rate. I've found that when MySQL is down, Drupal's Database::getConnection() function (in /[Drupal Root]/includes/database/database.inc) does not wait for a timeout; it fails immediately. So you have to deal with that:
kwynn_getConnectionWithRetry(); // defined just below
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);
function kwynn_getConnectionWithRetry() { // 2017/12/17 8:44pm EST
$sleep = array(2, 3, 5, 7, 17, 73, 181, 307, 601, 1201, 3607, 3697); // will retry several times over several hours
$sleepi = 0;
do {
$exception = false;
try {
// woe be unto you if you use anything other than 'default', 'default' -- see note below
Database::getConnection('default', 'default'); // a Drupal function defined in /includes/database/database.inc
} catch (Exception $exception) {
if ($exception->getCode() === 2002) sleep($sleep[$sleepi]); // see note on 2002 below
else throw $exception; // because if the problem isn't the database, there is no point in sleeping / waiting
}
} while($exception && isset($sleep[++$sleepi]));
}
I discovered the hard way that 'default', 'default' is hardcoded somewhere in Drupal. That's another discussion for some indefinite time.
You get code 2002 with both MySQL down and attempting to connect to an unreachable host / IP address. However the "message" is different. If the database is down, you get "SQLSTATE[HY000] [2002] No such file or directory". With an unreachable network, you get "SQLSTATE[HY000] [2002] Connection timed out".
Perhaps I should be adding more codes than 2002. I'll find out over the next few weeks.
You can simulate the timeout situation by setting your database "host" parameter to an unreachable address such as 192.168.0.30 if, on your network, that address is indeed non-existent or doesn't have MySQL.
When MySQL is down, you get:
PDOException object {
message => (string) SQLSTATE[HY000] [2002] No such file or directory
*Exception*string => (string)
code => (int) 2002
file => (string) /home/k/firm/includes/database/database.inc
line => (int) 321
...
When I first wrote about this error, I was the one who shut the database down. Weeks later, though, I had a related problem: a database timeout that led to the same "rebuild_theme" error.
At the beginning of 2017, I moved "my" branch of the Drupal code to Amazon Web Services (AWS). However, other developers were involved, so I had to leave the Drupal database on a non-AWS server. I suspect the Drupal code was never intended to be separated from the database, so I got 3 timeouts in 3 weeks. The Drupal code does not recover from a timeout, so I changed it so that it would.
My change is to [Drupal root]/includes/database/database.inc starting from Drupal 7.x in Github (raw) as of 2017/04; also see the GitHub Drupal parent page. (More specifically, I started from database.inc: df7b3fd [last updated on] on Sep 29, 2016 as designated in GitHub)
I made 2 changes. The first is centered around what used to be line 321:
// Kwynn 2017/02/12 - Creating a retry loop - 9:46pm
$driver_options[MYSQLI_OPT_CONNECT_TIMEOUT] = 160; // Kwynn - 160 seconds - the reason for 160 is somewhat arbitrary
$kwynnRTW = array(2, 3, 5, 17, 73, 181, 307, 601, 1201, 3607, 3697); // retry wait
$kwynnCnt = 0;
$kwynnIsEx = true;
do { // Kwynn doing retry
try {
// Kwynn note - PDO being "basic" PHP, not Drupal
// Call PDO::__construct and PDO::setAttribute.
parent::__construct($dsn, $username, $password, $driver_options); // Kwynn - originally line 321
$kwynnIsEx = false;
}
catch (Exception $ex) {
sleep($kwynnRTW[$kwynnCnt]); }
} while(isset($kwynnRTW[++$kwynnCnt]) && $kwynnIsEx); // END Kwynn fix
The first change was meant to apply to the first database connection (although I'm almost certain it's called from subsequent connections). However, I had to make the second change because the database connection would eventually timeout and I couldn't find a way to force Drupal to reconnect. This time, I changed (Drupal original lines) 1519 and 1533, as shown below. Further below I explain how to use this change from your module.
final public static function getConnection($target = 'default', $key = NULL, $force = false) { // Kwynn added force 2017/03/06
// [...]
if (!isset(self::$connections[$key][$target]) || $force) { // Kwynn added force
In my module, first I had to access the database.inc file "manually" because the relevant files were running before "drupal_bootstrap().":
require_once(dirname(__FILE__) . '/../../../../../../includes/database/database.inc');
Your path to Drupal root will likely be different, of course.
Then I created this function:
public static function checkAndReconnectDB() {
try { db_query("SELECT version()"); }
catch (Exception $ex) { Database::getConnection('default', 'default', true); }
}
Also, I have to call checkAndReconnectDB() at points in my code where I have reason to believe the database has timed out. Specifically, I was running an HTTP / curl timeout loop. Sometimes the HTTP server would reconnect but, by then, the database had timed out. So when the code leaves the HTTP retry loop, I run checkAndReconnectDB().
More specifically, I get this error when my database is down when I'm using command-line mode (PHP CLI), in Drupal7 / Drupal 7 / 7.51.
The full error is:
PHP Fatal error: Uncaught Error: Call to undefined function _system_rebuild_theme_data() in [Drupal root]/includes/theme.inc:798
Stack trace:
#0 [Drupal root]/includes/theme.maintenance.inc(57): list_themes()
#1 [Drupal root]/includes/bootstrap.inc(2898): _drupal_maintenance_theme()
#2 [Drupal root]/includes/errors.inc(179): drupal_maintenance_theme()
#3 [Drupal root]/includes/bootstrap.inc(2619): _drupal_log_error(Array, true)
#4 [internal function]: _drupal_exception_handler(Object(PDOException))
#5 {main}
thrown in [Drupal root]/includes/theme.inc on line 798
