研究・調査 RESEARCH
AskDona / GFLOPSが公開する長尺レポート・ベンチマークです。 Long-form research and benchmarks published by AskDona / GFLOPS.
リサーチ Research
論文を公開しました：LLMを用いた医療機器の文書アセスメントを、全工程で決定論的にする Preprint released: Making every stage of medical device document assessment with LLMs deterministic
GFLOPSは、医療機器の文書アセスメントにおけるLLMの再現性を検証した論文をarXivで公開しました。14の商用LLM環境の検証と、全工程を決定論的にしたパイプラインの結果を報告しています。 GFLOPS has released a preprint on arXiv about the reproducibility of LLMs in medical device document assessment. It reports tests of fourteen commercial deployments and a pipeline built to be deterministic at every stage.
Is SubQ Snap deterministic? We sent it the same request 30 times. Is SubQ Snap deterministic? We sent it the same request 30 times.
Byte-identical requests, no label flips, and probabilities that depend on whether the calls were sequential or concurrent. The same test we ran on Jev two weeks ago. Byte-identical requests, no label flips, and probabilities that depend on whether the calls were sequential or concurrent. The same test we ran on Jev two weeks ago.
Is Jev deterministic? We sent it the same request 30 times. Is Jev deterministic? We sent it the same request 30 times.
Byte-identical requests, a pinned model version, and two flipped labels. Why that matters for regulated assessment, and why we are making our whole pipeline deterministic. Byte-identical requests, a pinned model version, and two flipped labels. Why that matters for regulated assessment, and why we are making our whole pipeline deterministic.
LLMを用いたクエリ分類を実務で行うには：AskDonaチャットボットの数万件のクエリ分析から得られた実践的知見 LLM-based Query Classification Practices: Lessons from Analyzing Tens of Thousands of AskDona Chatbot Queries
本稿では、理化学研究所 計算科学研究センター（R-CCS）が運用する「富岳サポートサイト」および「スパコン成果ナビ」という2つの実稼働チャットボットを題材とし、LLMを用いたクエリ分類をどのように実施し、分類結果の品質を確保したのか、その具体的な手順と実施上の工夫を解説します。 This article uses two production chatbots operated by the RIKEN Center for Computational Science (R-CCS), the Fugaku Support Site and the Supercomputer Report Navigator, to explain how LLM-based query classification was implemented and how the quality of the resulting labels was maintained.
本番RAGにおけるStructural Blind Spots Structural Blind Spots in Production RAG
「精度N%」だけでは、本番RAGパイプラインの最も重要な評価軸——クエリクラスの構造的カバレッジ——を捉え損ねる。AskDonaの本番運用から得たStructural Blind Spotsの概略を示す。 Why accuracy alone cannot evaluate RAG pipelines — an outline of Structural Blind Spots, the class-level capability that 'N% accuracy' misses, drawn from AskDona production deployments at RIKEN R-CCS.