An administrative law judge may withdraw from a proceeding whenever he deems himself disqualified. Any party may request the administrative law judge, at any time following his designation and before filing of his decision, to withdraw on ground of personal bias or disqualification, by filing with him promptly upon the discovery of the alleged facts a timely affidavit setting forth in detail the matters alleged to constituate grounds for disqualification. If, in the opinion of the administrative law judge, such affidavit is filed with due diligence and is sufficient on its face, he shall forthwith disqualify himself and withdraw from the proceeding. If the administrative law judge does not disqualify himself and withdraw from the proceeding, he shall so rule upon the record, stating the grounds for his ruling and proceed with the hearing, or, if the hearing has closed, he shall proceed with issuance of his decision, and the provisions of Sec. 102.26, with respect to review of rulings of administrative law judges, shall thereupon apply. (49 Stat. 449; 29 U.S.C. 151-166, as amended by (61 Stat. 136; 29 U.S.C. Sup. 151-167), (65 Stat. 601; 29 U.S.C. 158, 159, 168), (73 Stat. 519; 29 U.S.C. 141-168), (88 Stat. 395-397; 29 U.S.C. 152, 158, 169, 183)) [45 FR 51193, Aug. 1, 1980]