上周四我們的IDC核心數(shù)據(jù)區(qū)到DMZ區(qū)的防火墻經(jīng)歷了十分詭異的故障。

雖然我從一開(kāi)始就斷定非ORACLE問(wèn)題，因?yàn)檫B接核心數(shù)據(jù)區(qū)的多個(gè)ORACLE數(shù)據(jù)庫(kù)的應(yīng)用都無(wú)一例外的出現(xiàn)了相同的問(wèn)題，如果三個(gè)以上ORACLE能同時(shí)出想這么嚴(yán)重的問(wèn)題，那么我想ORACLE也該“停業(yè)整頓”了。但是應(yīng)用的同事看得出來(lái)還是非常懷疑ORACLE出現(xiàn)了問(wèn)題，再加上鏈路可以PING通，最后通過(guò)替換法確定是防火墻問(wèn)題，總算告一段落。處理從當(dāng)天下午4：00一直持續(xù)到第二天上午8：00，可謂驚心動(dòng)魄。

故障現(xiàn)象：應(yīng)用報(bào)如下錯(cuò)誤：

SQL state [null]; error code [17410]; No more data to read from socket; nested exception is java.sql.SQLException: No more data to read from socket
java.sql.SQLException: No more data to read from socket
??????? at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
??????? at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
??????? at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:208)
??????? at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1118)
??????? at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
??????? at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:478)
??????? at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:216)
??????? at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:799)
??????? at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1037)
??????? at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:839)
??????? at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1132)
??????? at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3316)
??????? at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3361)
??????? at weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:100)
??????? at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:120)
??????? at org.hibernate.loader.Loader.getResultSet(Loader.java:1272)
??????? at org.hibernate.loader.Loader.doQuery(Loader.java:391)
??????? at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:218)
??????? at org.hibernate.loader.Loader.doList(Loader.java:1593)
??????? at org.hibernate.loader.Loader.list(Loader.java:1577)
??????? at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:395)
??????? at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:271)
??????? at org.hibernate.impl.SessionImpl.list(SessionImpl.java:844)
??????? at org.hibernate.impl.QueryImpl.list(QueryImpl.java:74)
??????? at cn.gov.zjport.persistence.core.hibernate.PersistenceHibernateImpl$1.doInHibernate(PersistenceHibernateImpl.java:96)
??????? at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:366)
??????? at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:334)
??????? at cn.gov.zjport.persistence.core.hibernate.PersistenceHibernateImpl.query(PersistenceHibernateImpl.java:93)
??????? at cn.gov.zjport.normandy.dao.persistence.base.impl.BaseDAOImpl.getListByFilter(BaseDAOImpl.java:109)
??????? at sun.reflect.GeneratedMethodAccessor466.invoke(Unknown Source)
??????? at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
??????? at java.lang.reflect.Method.invoke(Method.java:618)
??????? at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:287)
??????? at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
??????? at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
??????? at cn.gov.zjport.houston.service.base.cache.interceptor.ParaCacheInterceptor.invoke(ParaCacheInterceptor.java:68)
??????? at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
??????? at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
??????? at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
??????? at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
??????? at $Proxy595.getListByFilter(Unknown Source)
??????? at cn.gov.zjport.normandy.service.exchange.impl.ImbackDeclareServiceImpl.queryImbackDeclare(ImbackDeclareServiceImpl.java:65)
??????? at sun.reflect.GeneratedMethodAccessor465.invoke(Unknown Source)
??????? at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
??????? at java.lang.reflect.Method.invoke(Method.java:618)
??????? at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:287)
??????? at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
??????? at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
??????? at cn.gov.zjport.houston.service.base.cache.interceptor.ParaCacheInterceptor.invoke(ParaCacheInterceptor.java:68)
??????? at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
??????? at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
??????? at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
??????? at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
??????? at $Proxy602.queryImbackDeclare(Unknown Source)
??????? at cn.gov.zjport.normandy.service.webservice.ImbackDeclareServiceImpl.queryImbackDeclare(ImbackDeclareServiceImpl.java:57)
??????? at sun.reflect.GeneratedMethodAccessor464.invoke(Unknown Source)
??????? at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
??????? at java.lang.reflect.Method.invoke(Method.java:618)
??????? at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:410)
??????? at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
??????? at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:332)
??????? at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
??????? at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
??????? at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
??????? at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
??????? at org.apache.axis.server.AxisServer.invokeService(AxisServer.java:234)
??????? at org.apache.axis.server.AxisServer.invoke(AxisServer.java:375)
??????? at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:689)
??????? at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
??????? at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
??????? at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
??????? at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
??????? at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
??????? at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:283)
??????? at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
??????? at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
??????? at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:75)
??????? at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
??????? at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
??????? at com.opensymphony.webwork.dispatcher.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:88)
??????? at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
??????? at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3242)
??????? at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
??????? at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
??????? at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2010)
??????? at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:1916)
??????? at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
??????? at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
??????? at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
<Mar 3, 2011 8:37:25 PM GMT+08:00> <Error> <JDBC> <BEA-001112> <Test "SELECT 1 FROM DUAL" set up for pool "operation_8_zjport" failed with exception: "java.sql.SQLException: OALL8 is in an inconsistent state".>

究竟是什么問(wèn)題，還在調(diào)查中

轉(zhuǎn)載于:https://blog.51cto.com/yunlongzheng/509680

總結(jié)

以上是生活随笔為你收集整理的诡异的防火墙故障，能PING通，但访问ORACLE不稳定的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。